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ABSTRACT 



This report discusses the findings of a study that compared 
identified desirable characteristics of state accountability reports with 20 
accountability reports generated by 7 states. Desirable characteristics 
included (1) clarity (the intended audience, statements of purpose, a 
description of the population being reported on and the states ' conceptual 
model for its accountability system); (2) comprehensiveness in the reporting 
of inputs, processes, and results for students, especially students with 
disabilities; (3) provide comparative information with changes over time 
between schools, districts, states, regions, or standards; (4) conciseness or 
carefully chosen indicators so that no more information is given than is 
necessary; (5) cautions (against misinterpretations of the data or against 
any unintended consequences) ; (6) confidentiality; (7) good formats so that 

reports are well -organized, readable, and interesting. Results found none of 
the reports contained every desirable characteristic. Very few reports were 
clear as to who was included, the conceptual model, and assumptions. Many 
reports were comprehensive, concise, and comparative, but were also unclear 
in their presentation of information. Only four of the states provided 
disaggregated data on students with disabilities. (CR) 
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Executive Summary 



The Individuals with Disabilities Education Act (P.L. 105-17) was reauthorized in 1997, and 
one of the major additions to IDEA is the requirement that states report regularly on the perfor- 
mance and progress of all students, including students with disabilities. Most states now issue 
reports on student performance, and these accountability reports differ considerably in content 
and format. This study was conducted to identify stakeholders’ views of desirable characteris- 
tics of reports, and to investigate the extent to which the characteristics were evidenced in a 
diverse sample of seven states’ accountability reports. This study also provides guidelines for 
desirable characteristics in reporting that state department personnel can use to improve their 
accountability reports. 

A descriptive study of accountability reports showed that none of the states’ accountability 
reports that were examined met all of the desired characteristics. Four states reported disaggre- 
gated data on the performance of students with disabilities, yet these were primarily enrollment 
data. 

There are a number of improvements that can be made to increase the readability of educational 
accountability reports. The following are some of the recommendations: 

• Reports must contain clear statements of purpose, intended audience, a description of the 
population being reported on, their conceptual model, their mission/vision, and their 
assumptions. 

• Reports should be concise. One way to address this issue might be to provide a pyramid 
of reports (from a continuum of more basic to more detailed reports) for the various 
audiences. A parent or citizen may want a basic report while a researcher or school board 
member may want more detailed information. 

• Reports should be as uncomplicated as possible. Executive summaries, organizers such 
as a table of contents or glossary, and visual attention grabbers lik e pie graphs or picto- 
grams can help the reader understand the contents. 

By incorporating the guidelines in this study, we can improve communication on the educa- 
tional results for students, and work toward better education for all students, with and without 
disabilities. 
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Overview 



Increasingly, state education departments are recognizing that inclusive accountability systems 
and the public reporting of educational results for students are important tools to help students 
attain higher educational standards. Many states have already designed accountability systems 
to ensure those inside and outside the educational system that students are moving toward 
desired goals (Brauen, O’Reilly, & Moore, 1992). With recent federal mandates, such as P.L. 
105-17, the reauthorization of the Individuals with Disabilities Education Act (IDEA), state 
education agencies (SEAs) must report, with the same frequency they do for nondisabled students, 
the number of students participating in regular and alternate assessments. Furthermore, 
performance data for students with disabilities must be disaggregated in reporting of results for 
students (U.S. Department of Education, 1997). 

The development of accountability systems is complex and in no way does “one way fit all” 
(McCaul, 1993). Multiple stakeholders first clarify the values that they are trying to optimize, 
educate themselves about their choices and trade-offs they must make, and determine how their 
future must differ from the status quo. They then must determine a design, implement it, and 
set up a process for evaluating it until it works smoothly and they achieve their desired results. 

The creation of statewide accountability systems requires coordination (Benveniste, 1985). As 
accountability requirements have multiplied at the national and state levels, central school district 
administrators have had the responsibility for processing all accountability reporting 
requirements. Computerized management information systems permit the handling of large 
databases and schools can easily provide rapid reporting system information tied to the district 
information management systems (Benveniste, 1985). Integration of these data is also needed 
at the state level. Decisions about data gathering and distribution require coordination and 
integration. However, the usual organizational arrangement is for data to be collected by many 
different offices and agencies within a state government with no single body deciding what data 
to collect. Otherwise, as observed in previous reports on state assessment systems and reporting 
practices, one state can produce several accountability documents which come from different 
offices. New York is one example of a state that produces seven different accountability reports 
that come from various offices (Thurlow, Langenfeld, Nelson, Shin & Coleman, 1997). 

Current state practices for reporting information to various audiences are a function of what 
people say or think ought to be reported, tradition, or what other states are doing (e.g. “This 
office has always produced this document”). According to Thurlow et al. (1997), states vary 
greatly in their reporting practices. In this qualitative study, members of the National Center 
on Educational Outcomes (NCEO) found that almost every state does produce an accountability 
document, but that some states exclude students with disabilities from their accountability reports 
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altogether, others exclude those students who take tests with accommodations, while still others 
exclude those who have used a specific type of accommodation. 

State accountability reports vary considerably in format, length, level of information given, 
focus, and stated purposes (Thurlow et al., 1997). Some states produce five or six 500-page 
volumes annually, while others produce a two- to three-page report. Many states use tables, 
spreadsheets, and the Internet to communicate their educational results. A few states give only 
state level data; others give school, district, and state level data annually. These reports vary in 
their focus as many do not provide any outcome information on students with disabilities, 
especially in regards to test-based outcome data. For example, only 12 states report test-based 
outcome data on students with disabilities. Many states use these reports for accreditation 
purposes while others use them for technical assistance, diplomas, compliance with state 
requirements, or to generate local, district, and national comparisons. 

Performance standards should be clear, specific, and doable, so that parents, teachers, 
administrators, and the community can understand what they mean (LaPointe, 1996a; State 
Education Improvement Partnership, 1996). In trying to organize educational data, LaPointe 
(1996b) has also suggested that the key is to select a small, manageable number of indicators 
and to provide evidence of these same indicators for each school or each system, or each state, 
and demand that the observer take into account each of these sets of characteristics when making 
decisions. Consistency in measuring the same indicators is very helpful. At present, indicators 
used in accountability reporting cover a wide spectrum, from detailed financial information to 
student mobility rates, and from staffing information to minutes spent in math and reading. 
Some states report numbers of students who met state standards or goals; others do not. Further, 
every state had at least one report in which there were indicators that were not described clearly 
(Thurlow et al., 1997). 

It is extremely important to take into account the large number of factors that describe the 
context of learning and school (LaPointe, 1996b). Much recent literature has pointed to aspects 
of family behavior, the community, and even cultural characteristics that have been associated 
with students’ achievement outcomes (e.g., SES, family size, family involvement in school, 
parental education level, and caregiver-child relationships) (Bradley et al., 1994; Brody et al., 
1994; Campbell & Ramey, 1994; Caughy, DiPietro, & Strobino, 1994; Duncan, Brooks-Gun, 
Klebanov, 1994; Feldman & Walton- Allen, 1997; Greenwood, 1991). For example, Kirst(1990) 
reports that achievement studies rarely calculate the impact of socioeconomic status and 
environmental factors upon pupil achievement, and consequently, we cannot hold teachers 
accountable for factors that they are unable to influence. As we devise accountability reports 
and compare states or countries one to another, or examine the performance of students, contextual 
factors must be incorporated into our indicator systems. 



Without taking other contextual factors into consideration, there is a danger in inappropriately 
emphasizing results of national achievement test scores or incomplete “report cards.” Reliance 
on such measures may not permit evaluation of program effectiveness for students with 
disabilities (National Joint Committee on Learning Disabilities, 1991). Interviews with principals 
from the New American Schools Development Corporation (NASDC) regarding accountability 
reports confirmed that public reporting of accountability data through system or school report 
cards and by the press heightened their anxiety about narrowing the focus of public debate from 
their broad reform goals to things assessed by the system and profiled in these reports (Mitchell, 
1996). The attachment of rewards and sanctions to their accountability systems made principals’ 
concerns more pointed. In order to allay principals and other stakeholders’ concerns about the 
development of accountability systems, flexibility should be given to these stakeholders in 
developing such systems. 

Furthermore, NASDC principals in interviews highly recommended that you have to continue 
to take traditional accountability tests, but you also should look for new assessment instruments 
to test what you are trying to do because traditional tests do not do it justice (Mitchell, 1996). 
Additional indices or more thorough assessment methods may be needed to accurately evaluate 
the progress of students with disabilities (i.e., alternative assessment methods). Thus, the im- 
portance of using multiple measures in addressing accountability issues should be emphasized 
(LaPointe, 1996b). 



While states have made improvements in their performance indicators, these systems are only 
as good as their data base. Some crucial gaps remain. In many states, little data exist on middle 
schools. Not much is known about how tracks and courses in the middle grades determine 
academic choices in senior high schools. Little integration exists between colleges and 
elementary-secondary schools. Although postsecondary schooling options exist for students, 
data on how these postsecondary students perform in colleges are often not attended to. In 
many states, high school performance data focus primarily on those students bound for four- 
year colleges or on those in the bottom quartile. What are the outcomes for those students in the 
“middle tracks”? 



Kirst (1990) encourages states to closely scrutinize existing data. Possibly certain kinds of 
little-used financial data might be eliminated. New data demands on local SEAs might be 
eased by coupling them with reductions in other data requests. Creating a database that is 
manageable, concise but contains salient information will lead to improved accountability 
reporting practices and hopefully, improved educational outcomes. 



Studies have shown that the exclusion of students with disabilities in accountability systems 
can impact their education. Students who are left out of assessments, and out of accountability 
reports, tend not to be considered when reform efforts are being created and implemented (Leone, 
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McLaughlin, & Meisel; 1992). Often “what gets measured gets taught” as well as “who gets 
measured gets taught.” Exclusion can lead to educators’ beliefs that they are not responsible for 
the education of these students or lowered expectations for students with disabilities (Yell & 
Shriner, 1996). Further, the removal of special education students from the “accountability 
track” also results, to a large degree, in their removal from the “curriculum track,” and those 
learning expectations which guide the instruction of regular education students (Koehler, 1992). 
This results in the special education student becoming more and more isolated from the 
mainstream instructional program rather than having an alternate course being charted for 
reaching competence in mainstream subject area content. 

Failure to hold schools accountable for the outcomes of students with disabilities is ill advised 
when evidence exists that students with disabilities are not reaching satisfactory achievement 
in basic math and science concepts, school completion, and employment. Colorado’s Department 
of Education conducted a pilot study of indicators in several high schools (Colorado Department 
of Education, 1990). The study team analyzed data from 16 schools (939 students and 689 
regular and special education staff) using student records, student questionnaires and interviews, 
and questionnaires from regular education and special education teachers and administrators. 
Some disturbing results included higher absence and out-of-school suspension rates for students 
with disabilities. Half (50%) of the 12th grade special education students with emotional or 
behavior disturbances — or with mental retardation — did not receive a diploma or completion 
certificate. Those with high absence rates also reported lower levels of satisfaction with school 
and were less likely to experience positive expectations and support from their teachers (1990). 

Earlier studies of educational outcomes for students with disabilities have made special educators 
acutely aware that post-school outcomes of students with disabilities have not been satisfactory 
(Edgar, Levine, & Maddox, 1986; Hasazi, Gordon & Rose, 1985; Mithaug, Horiuchi & Fanning, 
1985). Data from the National Longitudinal Transition Study (NLTS; Wagner, 1991) have 
supported the findings of earlier studies that nationally, students with disabilities drop out of 
school at a much higher rate than their nondisabled peers and that only slightly more than half 
of the students who leave school do so via graduation. These results have caused educators, 
advocates and policymakers to call for better outcomes for students with disabilities. 

Not only should data on students with disabilities be reported, but data should also be 
disaggregated from the rest of the general education students. Better outcomes for students 
with disabilities are not assured even when their results are included in accountability reports; 
often the data can be overshadowed by the results of general education students (Geenen, Thurlow 
& Ysseldyke, 1995). Failure to report on disaggregated outcomes of students with disabilities 
leads to an absence of data on the effectiveness of special education services. McGill-Smith, an 
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advocate and parent, strongly links reform to the image of a train: 

Parents of children with disabilities are on the right platform, but we need tickets to get 
on the train. Outcomes data that help us evaluate reforms based on what works for our 
children are the tickets we need. (Ysseldyke, Thurlow, & Shriner, 1992, p. 49) 

By reporting educational results for all students, we will be better able to evaluate services and 
work towards better education for all students. 

Since current state reports vary greatly, are likely to be unclear, and often do not report 
disaggregated results for students with disabilities, this study was conducted to provide helpful 
reporting guidelines for state department personnel. A descriptive study of accountability reports 
was performed to address the following questions: (1) What do a group of stakeholders (state 
department personnel, special education representatives, assessment specialists) believe are the 
necessary, desirable, and succinct characteristics of good state and district educational 
accountability reports, and (2) To what extent do a sample of reports from seven diverse states 
reflect the criteria generated by this group of stakeholders, including criteria to include outcome 
data for students with disabilities? 

The reports chosen for this study were compiled before the passage of P.L. 105-17. Thus, this 
study provides baseline data on how states report educational results before the provisions of 
the reauthorized IDEA came into effect. This study also provides guidelines for desirable 
characteristics in reporting that state department personnel can use to improve their accountability 
reports. 



Methods — , ... = . 

Participants 

Under the leadership of the National Center on Educational Outcomes (NCEO), a group of 
state department personnel, special education representatives, and assessment specialists gathered 
in May, 1997 to discuss the necessary, desirable, and succinct characteristics of good state and 
district educational accountability reports. The participants in this special conference were 
members of the Council of Chief State School Officer’s (CCSSO) special task group entitled, 
“State Collaborative on Assessment and Student Standards Assessing Special Education Students 
Study Group #5” (SCASS study group). These members volunteered to serve on this committee 
at their national CCSSO meeting. See Table 1 for a list of nationally representative stakeholders 
who participated in the two-day conference. 
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Table 1. Participants in SCASS Focus Group 



Stakeholders 


Represented Organizations 


Margaret Crank 


Arkansas State Education Department 


Judy Elliott 


National Center on Educational Outcomes 


Ron Erickson 


National Center on Educational Outcomes 


Jim Friedebach 


Missouri Department of Education 


Marcia Harding 


Arkansas State Education Department 


Karen Langenfeld 


National Center on Educational Outcomes 


Vince Madden 


California Department of Education 


Ruth Nelson 


National Center on Educational Outcomes 


Ken Olsen 


Mid-Southern Resource Regional Center Director 


Ed Roeber 


Council of Chief State School Officers 


Dorene Scott 


National Center on Educational Outcomes 


Alan Sheinker 


Wyoming Department of Education 


Richard Smiley 


Alaska Department of Education 


Martha Thurlow 


National Center on Educational Outcomes 


Jerry Tindal 


University of Oregon 


Don Watson 


Colorado Department of Education 


Jim Ysseldyke 


National Center on Educational Outcomes 



Procedure 

The first day of the conference was devoted to describing what reports should look like for the 
various audiences such as policymakers, parents, researchers, the general public, teachers and 
administrators, students, and the media. The members, as a large group, brainstormed the 
necessary characteristics to include when designing good educational accountability reports. 
After brainstorming, the group described each characteristic and how it should be implemented 
into a report for parents, policymakers, or the general public. See Figure 1 for the Checklist of 
Guidelines for State and District Educational Accountability Reports. 

A nationally representative sample of seven states’ accountability documents (LA, KY, MI, MT, 
NY, OK, SC) were chosen to compare against the list of desirable characteristics generated by 
participants of the accountability conference. These states were not only geographically diverse, 
but varied greatly in their reporting practices as indicated in a previous study of all 50 states’ 
accountability documents (Thurlow et al., 1997). Accountability reports are public data published 
by state education departments and state education boards, and copies were obtained by 
telephoning state assessment directors named in the CCSSO annual survey (CCSSO, 1997). 
Since state documents are published by states at different times of the year, and many reports 
even within a single state can be produced at separate times, the author only used the most 
recent accountability documents published before December 1, 1997. A total of 20 accountability 
reports were gathered between August 1, 1997 and December 1, 1997. Refer to Appendix A for 
a complete listing of documents produced by the seven states. 
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Figure 1. Checklist of Guidelines for State and District Educational Accountability Reports 



Checklist of Guidelines for State and District Educational 
Accountability Reports 

CONTENT 

Clear... 

□ Clear statement of intended audience 

□ Clear statement of intended purpose 

□ Clear statement of states’ conceptual model for its accountability 
system (including inputs, processes, and results) 

□ Clear statement of state standards (or goals) or mission/vision 

□ Clear statement of assumptions 

□ Clear statement of who was included in the population of students 
being reported on 

Comprehensive... 

□ Comprehensive, yet concise set of inputs, processes, and results 

□ Data on all students, including students with disabilities and limited 
English proficient students (students with disabilities and limited 
English proficient students’ results are disaggregated) 

Comparative... 

Includes enough information to enable people to make fair 
comparisons among: 

□ Schools 

□ Districts 

□ States 

□ Regions 

□ Standards 

Includes enough information to enable people to make fair 
judgments about changes over time for: 

□ Schools 

□ Districts 

□ States 

Concise... 

□ Includes no more information than is necessary to convey a message 
to an intended audience ... brief 
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Figure 1. Checklist of Guidelines for State and District Educational Accountability Reports 
(continued) 



Cautions... 

Provides cautions against: 

□ Scapegoating 

□ Unintended consequences 

□ Negatives 

Confidentiality... 

□ Maintains confidentiality of low frequency student populations 



Readable... 

□ Appropriate for the intended audience 

Responsive to the needs of intended audiences... 

□ Answers audience questions and provides accurate profile 

Layout... 

□ Not cluttered or complex 

□ Organized and easy to find information (e.g., reader’s guide, table 
of contents, index, glossary) 

□ Interesting (e.g., includes catchy titles, pictures, or other devices to 
get and hold audience interest) 

Links... 

□ Statement of how and where to get additional copies 

□ Statement of how to get more detailed information 

Executive Summary... 

□ Bulleted summary of report for a “quick read” 



FORMAT 



IMPORTANT OVERALL QUESTIONS 



Is the report readable? 
Is the report fair? 



Is the report concise? 

Is the report visually attractive? 
Is the report accurate? 



□ YES □ NO 

□ YES □ NO 

□ YES □ NO 

□ YES □ NO 

□ YES □ NO 
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Each accountability document was read and compared against the Checklist of Guidelines to 
evaluate the reports. A Checklist was filled out for each of the document(s) produced by the 
selected states to see whether current state reporting practices reflect the criteria generated by 
the CCSSO study group. To ensure objectivity and accuracy in evaluating the reports, a reliability 
check was completed by another graduate assistant at the National Center on Educational 
Outcomes. The reliability index (agreements on selection of desired characteristics over 
agreements plus disagreements of selection of desired characteristics) was 93%. Frequency 
analyses of how many characteristics are included in state accountability reports were carried 
out and tabled. 
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Tables 2 and 3 contain descriptions of the content and format characteristics of the 20 reports 
examined in this study, and Table 4 indicates whether these reports met overall important 
characteristics. None of the reports of this sample had every characteristic provided by the 
Checklist; however, there were a number of reports that provided good examples of individual 
characteristics. 



Content 

The content characteristics included by the states in their reports are listed in Table 2. Only one 
state contained all of the content characteristics in one of their reports (NYCAR). (For a list of 
acronyms and their definitions, see Appendix A.) Of the 20 reports produced by the seven states 
in this study, only five reports (NY, OK, and SC) included at least four of six characteristics 
regarding the clarity of the purpose, mission/vision, intended audience of the report, state 
standards, assumptions, and description of the population being reported on. However, none of 
the reports provided a clear statement of a states’ conceptual model for its accountability system, 
and only one report (NY) contained a clear statement of assumptions about their accountability 
system. (One pilot study document from Iowa did mention that their accountability system is 
based on an adaptation of NCEO’s model of outcome domains. However, this document was 
not used in this study since it did not contain any data.) Finally, only 9 of the 20 reports (45%) 
had a clear statement of who was included in the population of students being reported on. 

A majority of states’ documents (N=16) were comprehensive; they included input, process, and 
outcome indicators and/or included data on all students, including disaggregated data on students 
with disabilities and limited English proficient students. Five of the seven states examined were 
considered comprehensive (having at least one of the two comprehensive characteristics) in 



O 

ERIC 



NCEO 



15 



9 



Table 2. Content Characteristics Included in the Educational Accountability Reports of Seven 
States 



State 


Clear 


Comprehensive 


Comparative 


Concise 


Cautions 


Confidentiality 


Iowa 














ACE 




■ 


■ 


■ 




NA* 


Kentucky 

KACR 




■ 


■ 


■ 




■ 


KIRIS 






■ 


■ 




NA* 


KCAAI 




■ 




■ 


■ 


NA* 


KSDL 






■ 


■ 


■ 


NA* 


Michigan 

MSR 




■ 


■ 


■ 


■ 


■ 


Montana 














MPE 








■ 




NA* 


MSS 








■ 


■ 


NA* 


New York 














NYCAR 


■ 


■ 


■ 


■ 


■ 


■ 


NYCSE 


■ 


■ 


■ 








NYSPE 




■ 


■ 




■ 


NA* 


NYSPP 




■ 


■ 






NA* 


NYPB 




■ 


■ 


■ 




NA* 


Oklahoma 














ODR 


■ 


■ 


■ 




■ 


■ 


OSR 


■ 


■ 


■ 


■ 


■ 


■ 


OSRC 




■ 




■ 




■ 


South Carolina 














SCSPP 




■ 


■ 


■ 




■ 


SCEP 




■ 


■ 


■ 




NA* 


SCSP 




■ 


■ 




■ 


■ 


SCPB 


■ 


■ 


■ 






NA* 



*NA = This report does not make any indication of confidentiality procedures, but fewer than five 
students in a population are not reported in the report. 
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Table 3. Format Characteristics of the Educational Accountability Reports of Seven States 



State 


Readable- 

Appropriate 
for Intended 
Audience 


Responsive... 

to Audience 
Needs 


Layout.. .is 

not Cluttered 
or Complex, 
but Organized 
and Interesting 


Links... to 
Get Additional 
Copies or More 
Detailed 
Information 


Executive 

Summary... 


Iowa 

ACE 


■ 


■ 


■ 






Kentucky 

KACR 












KIRIS 


■ 


■ 


■ 


■ 




KCAAI 












KSDL 












Michigan 

MSR 




■ 




■ 




Montana 

MPE 










■ 


MSS 


■ 




■ 




■ 


New York 
NYCAR 




■ 




■ 




NYCSE 




■ 


■ 


■ 


■ 


NYSPE 


■ 


■ 


■ 


■ 


■ 


NYSPP 












NYPB 


■ 


■ 


■ 


■ 




Oklahoma 

ODR 


■ 


■ 


■ 


■ 




OSR 


■ 


■ 


■ 


■ 


■ 


OSRC 


■ 




■ 


■ 




South 

Carolina 

SCSPP 












SCEP 


■ 


■ 


■ 


■ 




SCSP 




■ 








SCPB 


■ 


■ 


■ 




■ 
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Table 4. Important Overall Questions of the Educational Accountability Reports of Seven 
States 



State 


Is the 

Report 

Readable? 


Is the 

Report 

Fair?* 


Is the 

Report 

Concise? 


Is the 
Report 
Visually 
Attractive? 


Is the 

Report 

Accurate?* 


Iowa 

ACE 


■ 


■ 


■ 


■ 


■ 


Kentucky 

KACR 




■ 


■ 




■ 


KIRIS 


■ 


■ 


■ 


■ 


■ 


KCAAI 




■ 


■ 




■ 


KSDL 




■ 


■ 




■ 


Michigan 

MSR 




■ 


■ 




■ 


Montana 

MPE 




■ 


■ 




■ 


MSS 


■ 




■ 




■ 


New York 
NYCAR 




■ 


■ 




■ 


NYCSE 




■ 




■ 


■ 


NYSPE 


■ 


■ 




■ 


■ 


NYSPP 




■ 






■ 


NYPB 


■ 


■ 


■ 


■ 


■ 


Oklahoma 

ODR 


■ 


■ 




■ 


■ 


OSR 


■ 


■ 


■ 


■ 


■ 


OSRC 


■ 


■ 


■ 


■ 


■ 


South 

Carolina 

SCSPP 




■ 


■ 




■ 


SCEP 


■ 


■ 


■ 


■ 


■ 


SCSP 




■ 






■ 


SCPB 


■ 


■ 




■ 


■ 



■ = Yes 

* Judgments about fairness and accuracy were subjective and based only on the written reports that 
were reviewed. 
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their reporting practices. Four of the seven states (IA, MI, NY, SC) included disaggregated data 
of students with disabilities. Eleven of the twenty reports (55%) included disaggregated data on 
students with disabilities. However, these reports primarily contained enrollment data. Two 
states (NY and SC) did provide disaggregated test-based data for students with disabilities. Two 
of New York’s five reports were separate reports for students with disabilities. 

Sixteen of the 20 reports (80%) provided comparative information such as comparisons between 
schools, districts, states, regions, or standards and over time for a particular state. All the states 
provide some comparative information in at least one of their reports. Even the two page report 
provided by Montana used normal curve equivalents to compare students with a “norm” that 
had been established at the broad national level (MSS). Four states provided at least one indicator 
that was compared to a national average (IA, MT, NY, SC). Many of these reports included 
comparative information between districts and schools (i.e., the same information was printed 
for each district and school and conclusions could be drawn with cautions). South Carolina also 
provided comparisons between districts on the performance on standards in two of its reports 
(SCSP and SCPB). Finally, the majority of reports contained at least two to five years of data to 
use to make comparisons and to note trends for districts and states. 

Of the 20 reports reviewed, 14 (70%) were concise or did not provide more information than 
was necessary. Every state produced at least one report that was concise. These reports ranged 
in page length from a two page report (MSS) to a state that produced a 35 page state report 
(ODR). Of these 14 reports that were concise, only three were clear as well. Eleven of the 
fourteen reports that were concise were considered comprehensive. 

In providing cautions within their educational accountability reports, only nine state reports 
(45%) provided cautions to be careful to: (1) not judge effectiveness of an educational program 
based on one test score, (2) remember the many other contextual factors that may determine a 
student’s performance, (3) look at scores based on small numbers of students as not less reliable 
and valid, (4) note that school district area changes and service area changes may have occurred, 
and (5) realize there may be missing information that was not reported. 

Confidentiality was noted if a state provided a footnote or remark in the text that they were 
careful not to report any scores based on populations of fewer than five to ten people. Many 
state reports did not provide such a note, but also did not provide scores for fewer than five 
students (N=ll). Eight reports included such notes regarding confidentiality; however, one 
state report included results for populations of one student (NY CSE, the separate special education 
report). 
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Format 



Only 50% of the reports (N=10) were considered readable (glossaries provided, readable type) 
or appropriate for the audience needs; 55% (N=l 1) had layouts that were not cluttered or complex 
(see Table 3). Twelve of the twenty reports were responsive to audience needs (60%). Only 
50% of the reports provided links to get additional copies or more detailed information. Further, 
only six reports (30%) provided the reader with an executive summary of the report. Finally, 
two reports had all of the format characteristics as designated by the Checklist of Guidelines for 
State and District Educational Accountability Reports (NYSPE, OSR). Six reports had four of 
the five format characteristics (KIRIS, NYCSE, NYPB, ODR, SCEP, SCPB). 



Important Overall Questions 

As stated before, only 50% of the reports were considered readable, and only 50% of the reports 
were visually attractive (e.g., they included objects and devices that would hold the attention of 
the reader such as bar graphs, pictographs, balloons describing where to get more information, 
etc.) (see Table 4). Many of the hard to read and visually unattractive reports included spreadsheet 
formats with a large amount of information (numbers in small font) per page with short labels 
for categories. Nineteen of the twenty reports were considered fair. The one report that was not 
marked as fair was from Montana, in which scores were reported in normal curve equivalents 
based on five different achievement tests. The twenty reports appeared to be accurate. Six of the 
twenty reports were considered readable, fair, concise, visually attractive, and accurate (ACE, 
KIRIS, NYPB, OSR, OSRC, SCEP). 



Discussion ■ « ... »=■ =■== * ■= ... . = a , 

Desired characteristics of state accountability reports were developed by primary stakeholders 
(state department personnel, special education representatives, assessment specialists), and the 
extent to which a sample of seven states’ reports reflected those characteristics were examined. 
Though only 20 documents from 7 diverse states were sampled, it appeared that none of the 
reports contained every characteristic desired by primary stakeholders. Especially disconcerting 
was the result that very few reports (25%) were clear as to who was included in the population 
of students being reported on, the conceptual model used for its accountability system, and the 
assumptions about the state accountability systems. There were a number of reports that were 
comprehensive, concise, and comparative, but were also unclear in their presentation of 
information. A state can report a lot of data, but if it is not clear what the purpose of the report 
is, who it is intended for, what the conceptual model of accountability is, or who was included 
in the populations being reported on, the report can become meaningless to stakeholders. 
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Similarly, if only half of the documents were readable or appropriate for audience needs, then 
these reports will not be used by the audiences. For example, those reports that contained many 
pages of spreadsheet data with several undefined acronyms will not be understood by a typical 
parent, school board member, a school administrator, or university researcher. 

Four of the seven states provided disaggregated data on students with disabilities. This is 
comparable to Thurlow et al.’s (1997) study that found 30 states reported at least one indicator 
of disaggregated data on students with disabilities. However, these data were primarily 
enrollment, and not outcomes data. Only two states report test-based disaggregated data for 
students with disabilities. We acknowledge that the nonrandom selection of seven states prohibits 
national generalizations. Having reviewed and completed a qualitative study on all 50 states’ 
documents recently, we chose states that represented the diversity of reporting practices currently 
being used. 

The instrument (Checklist of Guidelines for District and State Accountability Reports) was 
compiled as a tool for state department personnel to use when creating their accountability 
documents. It is only a checklist, with items that could be considered subjective. For example, 
the items asking, “Is this report accurate? Is this report fair?” are highly subjective because 
judgments can only be based on what appears to be accurate and fair. If a state department 
director were to fill this form out, she/he would know whether the data were accurate (missing 
data, deleted test forms and scores, all students were included, and so forth). It may even be 
presumptuous for a researcher to make those types of determinations based only on written 
educational reports. Furthermore, these results may not reflect actual practice. We knew from 
prior knowledge that one of the states did include students with disabilities in its score results, 
but nowhere in the text of the report was it clear who was actually being reported on, and thus 
was marked as being unclear. In order to prevent subjective bias of the one rater, another rater 
was employed to conduct a reliability check. However, this person also had previous experience 
with accountability reports, which may have affected their decisions. 

These cautions aside, this study does provide baseline data on how states report their educational 
results before the provisions of the reauthorization of IDEA came into effect. Now that states 
will be required to provide alternate assessments and report disaggregated data on students with 
disabilities, reporting practices hopefully will change and become more inclusive. 
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Policy Implications and Future Directions 



A number of policy implications can be drawn from this study. Most importantly, outcome data 
must be disaggregated for students with disabilities. This study confirmed again the paucity of 
information on students with disabilities, especially outcome data (Thurlow et al., 1997; 
Y sseldyke, Thurlow, Langenfeld, Nelson, Seyfarth, & Teelucksingh, 1998). If we are not 
reporting on students with disabilities, then how are we paying attention to the educational 
processes used with these students? 

There are a number of improvements that can be made to increase the readability of these 
educational reports. Foremost, reports must contain clear statements of purpose, intended 
audience, a description of the population being reported on, their conceptual model, their mission/ 
vision, and their assumptions. If creators of a report know that it is intended for parents, then 
they will create a document that is more appropriate to their needs and background knowledge. 

Conciseness is an issue when it comes to reporting. Many documents provide hundreds of 
pages of data. Is this necessary? Should we select indicators more carefully? Are these large 
documents really being used in their entirety? One way to address this issue might be to provide 
a pyramid of reports (from a continuum of more basic to more detailed reports) for the various 
audiences. A parent or citizen may want a basic report while a researcher or school board member 
may want more detailed information. This pyramid can lead to efficient reporting practices as 
well as more pointedly target various audiences’ information needs. 

Reports should be as uncomplicated as possible. Executive summaries, organizers such as a 
table of contents or glossary, and visual attention grabbers such as pie graphs or pictograms can 
help the reader understand the contents. Additionally, keeping the number of concepts per page 
limited will increase the understandability and readability of the reports. Finally, reports should 
provide links to additional copies or further detailed information. 

Now that schools are responsible for reporting the educational outcomes for all students, it is 
important to examine how accountability reports are presented. Reports should especially be 
meaningful and presented in a clear format. By incorporating the above guidelines compiled by 
stakeholders, we can improve communication on the educational results for students, and work 
toward better education for all students, with and without disabilities. 
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List of Acronyms and State Accountability Documents Used in Analyses 
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Acronyms and State Accountability Documents 



Iowa 

ACE. Iowa Department of Education (1997). The annual condition of education report: A 
report on prekindergarten, elementary and secondary education in Iowa. Des Moines, IA: Author. 



Kentucky 

KACR. Kentucky Department of Education (1996a). Kentucky Instructional Results Information 
System 1995-96 assessment curriculum report. Frankfort, KY: Author. 

KIRIS. Kentucky Department of Education (1996b). Kentucky Instructional Results Information 
System individual student report. Frankfort, KY: Author. 

KCAAI. Kentucky Department of Education (1997a, Sept.). Kentucky school and district 
accountability results: Accountability cycle 2 (1992-93 to 1995-96) of content area academic 
index grades 4, 8, 11/12. Frankfort, KY: Author. 

KSDL. Kentucky Department of Education (1997b, Sept.). Kentucky school and district 
corrected accountability results: Accountability cycle 2 (1992-93 to 1995-96) of school and 
district listings. Frankfort, KY: Author. 



Michigan 

MSR. Michigan Department of Education (1996, March). Michigan school report [On-line]. 
Available: World Wide Web: http.V/www.mde.state.mi/ 



Montana 

MPE. Montana Office of Public Instruction (1997a). Montana public school enrollment data: 
Fall 1996-97. Helena, MT: Author. 

MSS. Montana Office of Public Instruction (1997b, April). Montana statewide summary: Norm- 
referenced student assessment reporting. Helena, MT: Author. 
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NYCAR. University of the State of New York & New York State Education Department (1996a, 
Dec.). A guide to the comprehensive assessment report of the New York state pupil evaluation 
program: 1996 Edition. Albany, NY: Author. 

NYCSE. University of the State of New York & New York State Education Department, Office 
of Vocational and Educational Services for Individuals with Disabilities (1996b, May). 
Consolidated special education performance report for 1994-1995. Albany, NY: Author. 

NSPE. University of the State of New York & New York State Education Department (1997a, 
Feb.). A report to the governor and the legislature on the educational status of the state s schools: 
Statewide profile of the educational system. Albany, NY: Author. 

NYSPP. University of the State of New York & New York State Education Department (1997b, 
Feb.). A report to the governor and the legislature on the educational status of the state ’s schools: 
Statistical profiles of public school districts. Albany, NY: Author. 

NYPE. University of the State of New York & New York State Education Department, Office 
of Vocational and Educational Services for Individuals with Disabilities (1996d, Aug.). 1996 
VESID Pocketbook of goals and results for individuals with disabilities. Albany, NY: Author. 



Oklahoma 

ODR. Oklahoma Department of Education, Education Oversight Board & Office of 
Accountability (1997a, June). Profiles 1996: Oklahoma educational indicators program district 
report volume I - eastern Oklahoma. Oklahoma City, OK: Author. 

ODR. Oklahoma Department of Education, Education Oversight Board & Office of 
Accountability (1997b, June). Profiles 1996: Oklahoma educational indicators program district 
report volume II - western Oklahoma. Oklahoma City, OK: Author. 

OSR. Oklahoma Department of Education, Education Oversight Board & Office of 
Accountability (1997c, April). Profiles 1996: Oklahoma educational indicators program state 
report. Oklahoma City, OK: Author. 

OSRC. Oklahoma Department of Education, Office of Accountability (1996). 1995-96 School 
report card, Edmond District, Santa Fe High School. Oklahoma City, OK: Author. 
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South Carolina 



SCSPP. South Carolina Department of Education (1996a). School performance profile 1996. 
Columbia, SC: Author. 

SCEP. South Carolina Department of Education (1996b). South Carolina education profiles. 
Columbia, SC: Author. 

SCSP. South Carolina Department of Education (1996c). State performance profile 1996. 
Columbia, SC: Author. 

SCPB. South Carolina Department of Education (1996d). What is the penny buying for South 
Carolina? Columbia, SC: South Carolina State Board of Education. 
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